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1. Introduction 


The District of Columbia (DC) Opportunity Scholarship Program (OSP) is the only federally 
funded program that provides vouchers to low-income families to send their children to private schools. 
Congress created the OSP in 2004 and reauthorized it in 2011 under the Scholarships for Opportunity and 
Results (SOAR) Act.' As part of the 2011 SOAR Act, Congress required an independent evaluation of the 
OSP. This is the sixth’ and final report from that evaluation, describing how the OSP affected students 
and their parents three years after they applied to the program. Specifically, the report examines impacts 
on student achievement, student and parent satisfaction with schools, student and parent perceptions of 
school safety, and parent involvement with education. 


Overview of the Program 


The SOAR Act establishes criteria for student Exhibit 1. Key elements of the 
eligibility, the groups of students who receive priority Opportunity Scholarship 
Program as defined in the 
SOAR Act 


Student eligibility criteria 
e DC resident 


for scholarships, and scholarship dollar amounts 
(exhibit 1). A program operator administers the OSP 
through a grant awarded by the U.S. Department of 


Education. Program operators establish protocols for e Income at or below 185 percent of the 
families applying to the program, recruit applicants federal poverty line at application 
and schools, award scholarships, and place and Priority given to students who: 

: . : itn coe : Had a sibling already in the program 
monitor scholatsiup awardees in partepaliie private Meade alow neviarming schcolinmced 
schools (see appendix A-1 for more information). of improvement 
Participating private schools must agree to Were offered a scholarship in the past but 
requirements regarding nondiscrimination in Gide ase 

q . 8 8 es j ; Were not already taking advantage of 
admissions, fiscal accountability, having teachers with school choice 
at least a bachelor’s degree, and cooperation with an Initial scholarship amount 
evaluation of the program. ¢ $8,000 for grades K-8 


e $12,000 for grades 9-12 


Overview of the Evaluation 


Congress required the evaluation to use “the strongest possible research design” to measure the 
impacts of being offered and using an OSP scholarship on key outcomes (exhibit 2).? To determine the 
OSP’s effectiveness, an experiment—considered the “gold standard” of evaluation methodology—was 
conducted that compared outcomes for two groups. The treatment group was comprised of students who 
applied for a scholarship and were offered one. The control group was comprised of students who applied 


' See http://www.gpo.gov/fdsys/pkg/BILLS- 1 12hr471eh/pdf/BILLS-112hr471eh.pdf for the SOAR Act legislation. The program was 
reauthorized in the Omnibus Reconciliation Act for 2017 spending, H.R. 244. 

? Congress required annual reports from the evaluation. The first three described the characteristics of program applicants and participating 
schools, parents’ considerations in applying to the OSP, and how participating schools differed from traditional public and charter schools in DC 
that OSP applicants might be able to attend. The fourth and fifth reports described the impacts of the OSP one and two years after families 
applied to the program. Reports are available at: https://ies.ed.gov/ncee/projects/evaluation/choice_soar.asp 

> Section 3009 of the SOAR Act also required the evaluation to examine retention, high school graduation, and college admission rates. However, 
because the majority of the evaluation’s sample was in elementary school (see figure 1) these outcomes were not examined in this report. 
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for a scholarship but were not offered one. Lotteries 


Exhibit 2. Key evaluation questions 


were used to randomly award scholarships to 


applicants. Randomization helped to ensure that the Tee Reading and Mathematics Achievement 


What is the effect of receiving/using an OSP 


two groups being compared were truly similar at the PRO Leer ipronrreadine saucnnine mic: 


time of OSP application, and that—other than by achievement? 
chance—the only difference that could influence Wc atisticuan 
outcomes was whether applicants received a What is the effect of receiving/using an OSP 


scholarship offer or not. scholarship on parent and student satisfaction 
with the student’s school? 


The evaluation included: - School Safety 
What is the effect of receiving/using an OSP 
scholarship on parent and student 


e = = An initial study sample of 1,771 eligible pereontionsiof school eatery? 


apDRCAnts of which 56 parount were offered Bare leealvemene 
scholarships.” Program applicants participated What is the effect of receiving/using an OSP 


in lotteries conducted in spring or summer of scholarship on parent involvement in their 
2012, 2013, and 2014. When they applied to the child’s education at home and at school? 
OSP, most were young (68 percent were 


entering elementary grades K—5, figure 1), low performing (scoring at the 34th percentile in 
reading and the 32nd percentile in mathematics nationally), and attending a school in need of 
improvement (64 percent). With the exception of those who were in pre-Kindergarten at 
application (25 percent), applicants were split between previously attending traditional public 
schools (40 percent) and charter schools (36 percent). 


Figure 1. Percentage of eligible applicants, by entering grade level at time of application 


Percent 
30 


24.9 
20 
11.1 
9.5 
10 iia 8.2 8.1 £7 
5.9 5.8 ae = 
2 3.6 
2.1 
Tiinie = 
0 mm 

K 1 2 3 4 5 6 7 8 9 10 11 12 
NOTE: Percents may not add to 100 because of rounding. 
SOURCE: OSP applications. 


4 As expected, the characteristics (including achievement) of the treatment and control groups when they first applied to the OSP were similar. 
For example, average reading scores at the time of application were 561 for the treatment group and 563 for the control group. Appendix A-3 
provides additional detail on the study’s sample. 
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e = Use of a nationally normed achievement assessment, as required in the SOAR Act. Students were 
tested both when they applied to the OSP and each spring for three years afterwards in the schools 
they were attending.° Students and parents were also surveyed each spring for three years, and 
principals were surveyed each year of data collection. Because families moved (sometimes out of 
DC) or refused to participate in data collection, and students were sometimes absent when the 
study team conducted testing, the analysis of program impacts each year was based on a subset of 
the initial study sample—those who provided data that year. 


e Standard approach to estimating impacts of being offered and of using a scholarship. The 
impacts of a scholarship offer were based on a comparison of outcomes of the treatment group and 
outcomes of the control group, taking into account students’ achievement and student and parent 
characteristics at the time of application (see appendix B-3 for more information). However, 

22 percent of students in the treatment group did not use a scholarship within the first three years,° 
and it is unlikely that these students were affected by the program. Therefore, the evaluation also 
estimated the impacts of using a scholarship.’ Both analyses included the students and parents who 
provided data in the third year (63 to 70 percent, depending on the outcome being measured), and 
used weighting adjustments to account for those who did not respond to surveys or complete 
testing.® 


e Impacts estimated for the whole sample and 
Exhibit 3. Impacts were measured for 


; . the following student 
examined program impacts for students groups: 


for specific student groups. The study 


: : r 
attending schools in need of improvement all ctadenis 


eee eeeaen cestenated igs ; Students attending or not attending a school 

group of students as a priority for scholarship in need of improvement 

awards. To explore hypotheses about how the Students scoring above or below the median 

OSP works and for whom, and to be consistent in reading 

with the previous evaluation of the program Students scoring above or below the median 
: in mathematics 

(Wolf et al., 2010), the study also examined 


impacts for several other subgroups of students 


Students entering elementary grades (K—S5) 
or secondary grades (6—12) 


(exhibit 3), as measured at time of application. 


° PL 112-10, Sec. 3009(a)(2)(B)(i) requires the evaluation to measure the impact of the program on student achievement, and Sec. 3009(a)(3)(A) 
requires the use of a norm-referenced standardized test. The study administered reading and mathematics tests to students in grades K-12 from 
the TerraNova Third Edition. These tests are vertically aligned and thus able to accurately capture growth in learning as students move from 
grade to grade (see appendix section B-5 for more information about the test). 

° Among the third-year impact sample of treatment group students, about half (49 percent) used the scholarship consistently across all three years, 
while others used it only in some years (29 percent) or not at all (22 percent) (appendix figure A-5). Three years after applying, 62 percent were 
using their scholarship. 

’ The study used a straightforward adjustment procedure attributed to Bloom (1984), which involved dividing the intent-to-treat impact by the 
proportion of students who used scholarships (see appendix section B-3 for more information). 

8 The analysis in this report was based on the 69 percent of students who completed tests in reading and the 68 percent who completed tests in 
mathematics, the 63 percent of students who completed the student survey, and the 70 percent of parents who completed the parent survey during 
the third year of followup data collection (see appendix B-5). These response rates were typical for studies that test students and survey parents 
over a number of years. However, to estimate impacts for the program overall and not just for those who provided data in the third year, the study 
constructed nonresponse weights. These weights align the characteristics of responding students and parents to the characteristics of all students 
and parents at the time of application and are used for the impact analyses (see appendix B-7). 

° Schools were categorized as “in need of improvement” by DC’s Office of State Superintendent for Education as part of the existing 
accountability provisions under federal law. 
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2. Impacts on Key Outcomes 


Impacts on Reading and Mathematics Achievement 


Improving academic achievement is a clearly stated goal of the SOAR Act. The legislation noted 
that public school students in DC perform well below national averages on reading and mathematics tests, 
and gave priority in the OSP to serving students who attend schools in need of academic improvement. 


Average test scores for the treatment and control groups are presented in this report as percentiles, 
a more understandable metric than the scale scores used in impact analyses. '° 


There were no statistically significant impacts on either reading or mathematics achievement 
three years after students applied to the program. Students in the group that received a scholarship 
offer scored 0.1 percentile points higher on the mathematics test, and 1.6 percentile points lower on the 
reading test, than students in the control group (figure 2) after three years. Students using a scholarship 
scored 0.2 percentile points higher on the mathematics test, and 2.1 percentile points lower on the reading 
test, than students in the control group. None of the differences were statistically significant. 


Figure 2. Impacts on reading and mathematics achievement (percentile scores) for 
scholarship offer and use, in third year 


Percentile @ Treatment 


= Control 
100 


80 Reading Mathematics 


60 Impact: 


-2.1 
40 


Scholarship Scholarship Scholarship Scholarship 
offered used offered used 


NOTE: Sample size was 571 treatment group students and 366 control group students for reading, and 569 treatment group 
students and 365 control group students for mathematics. 

SOURCE: Estimated means and impacts were generated from the study’s regression models, as described in appendix 
section B-3. Percentiles were calculated using grade-level norms and scale scores. The study administered the TerraNova 
Third Edition reading and mathematics tests to students participating in the OSP evaluation, three years after application. 


'0 For example, a student scoring at the 34th percentile performed better than 33 percent of students taking the test nationally, which is more 
meaningful than reporting an average scale score of 598. The study’s models estimated impacts using scale scores. Appendix section B-4 
provides detail on how percentiles were computed. 
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There were no statistically significant impacts on either reading or mathematics achievement 
for students in any of the study’s eight subgroups. In each case, those offered or using a scholarship 
had test scores that were similar to those not offered a scholarship (appendix tables C-1 and C-2). This 
included (1) students attending schools in need of improvement when they applied to the OSP and 
students not attending schools in need of improvement, (2) students entering elementary and secondary 
grades when they applied, (3) students scoring above or below the median!! in reading at the time of 
application, and (4) students scoring above or below the median in mathematics at the time of application. 


Impacts on Chronic Absenteeism 


Although absenteeism is not an outcome Congress asked the evaluation to examine, it is closely 
related to ones that were—achievement and high school completion. Students chronically absent during 
the early elementary years are less likely to be proficient in mathematics and reading by third grade and 
students chronically absent in the later grades are at risk for dropping out of school before graduation 
(Mac Iver & Mac Iver, 2010; Gottfried, 2011; Gottfried & Kirksey, 2017; Roby, 2004). Increasingly, 
states are using rates of chronic absenteeism as a measure of school quality under the Every Student 
Succeeds Act. Chronic absenteeism commonly is defined as a student being absent for 10 percent or more 
of the school year, about one month of school (Bauer et al., 2018). 


The study collected student attendance data from private schools and the DC Office of the State 
School Superintendent (OSSE).'? The number of days absent was converted to the percentage of the 
school year absent by dividing by the number of days in the school year.'? 


Students who were offered or used an OSP scholarship were less likely to be chronically 
absent three years after applying to the program. Students in the group that received a scholarship 
offer had a rate of chronic absenteeism that was 5.4 percentage points lower than the group not offered a 
scholarship (21.9 percent compared to 27.3 percent) (figure 3). Students using a scholarship had a rate of 
chronic absenteeism that was 7.5 percentage points lower than the control group. The differences were 
statistically significant. 


The OSP’s impacts on absenteeism were significant for students who were older and initially 
lower performing. Among students in grades 6—12 when they applied to the program, those offered 
scholarships had chronic absence rates 15.2 percentage points lower than students in the control group. 
For students with reading scores below the median when they applied, chronic absence rates were 
10.1 percentage points lower for students offered scholarships. Differences for the other six subgroups 
were not statistically significant (appendix table C-3). 


"' Defined in relation to the median performance of study participants at the time of application. 

"2 Due to availability of data, absenteeism was examined for this final impact report only. The study defined students as absent if they were absent 
the entire day, regardless of whether the absence was excused or unexcused. It is important to note that there were likely differences both across 
sectors (public vs. private) and even within sectors (among private schools or among public schools) in attendance policies and how consistently 
they were applied. 

'8 The denominator was the number of days in the school year of the specific school a student attended at the time of testing. The study also 
measured impacts on the percentage of school days absent, which are reported in appendix section D-6. 
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Figure 3. Impacts on student chronic absenteeism for scholarship offer and use, in third 
year 


Percent 
100 


@ Treatment 
@ Control 
80 


60 


40 ; -7.5* 


oes eal 


Scholarship Scholarship 
offered used 


*Difference between the treatment group and the control group was statistically significant at the 0.05 level. 

NOTE: Sample size was 585 treatment group students and 460 control group students. 

SOURCE: Estimated means and impacts were generated from the study’s regression models, as described in appendix section 
B-3. Student attendance data from the Office of State Superintendent for Education and private schools for school years 
2014-15, 2015-16, and 2016-17. 


Impacts on Parent and Student Satisfaction 


The OSP legislation called for the study to look at parent and student satisfaction with the child’s 
school. Recent research suggests that parents are more satisfied if they choose their child’s school 
(Barrows et al., 2017; Grady & Bielick, 2010). That research also indicates high levels of parent 
satisfaction regardless of school type, though there is some evidence that parents with children in private 
schools report the greatest satisfaction. This study compared satisfaction levels of parents and students in 
the treatment group with those in the control group, where a majority of families in both groups have 
exercised school choice. In DC, families have the option of applying through a common lottery to attend 
public schools other than their assigned neighborhood school. Three years after applying to the program, 
the percentage of students attending a charter school, private school, or traditional public school other 
than their assigned neighborhood school was 91 percent for the treatment group and 78 percent for the 
control group. 


The study administered surveys annually to parents and students in grades 4-12 to gauge 
satisfaction with the school that students were attending. For the primary measure of satisfaction, parents 
and students were asked to grade the school using a range from A to F. For this analysis, parent and 
student responses that gave the school a grade of A or B were grouped into one category and all other 
responses were grouped into the other category. 


The program had no statistically significant impact on parents’ satisfaction with their child’s 
school, but had a positive impact on students’ satisfaction three years after they applied to the 
program. The proportion of parents giving their child’s school an A or B was similar for parents of 
students offered the scholarship (81.6 percent) and parents of students not offered (80.5 percent) 
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(figure 4).'* Scholarship use also had no statistically significant impact on parent satisfaction. However, 
students’ satisfaction was 8.4 percentage points higher, with 68.5 percent of students offered the 
scholarship giving their school an A or B compared with 60.1 percent of students not offered the 
scholarship; the difference was statistically significant. The positive impact of scholarship use on 
students’ satisfaction was 11.0 percentage points. Appendix section D-2 provides additional information 
for interpreting the student survey results, which had relatively low response rates. 


There was little evidence that the program improved school satisfaction for individual 
subgroups of parents or students. There were no statistically significant impacts on school satisfaction 
for any subgroup of parents (appendix table C-4). For student satisfaction, there was a statistically 
significant impact for only one subgroup, students with reading scores above the median. For each of the 
remaining seven subgroups, findings were not statistically significant (appendix table C-5). 


Figure 4. Impacts on parent and student satisfaction (percentage giving school an 
A or B grade) for scholarship offer and use, in third year 


@ Treatment 


Percent Parent Student = Control 


100 Impact: Impact: 
1.0 1.3 Impact: Impact: 


8.4* 11.0* 


80 
60 
40 


Scholarship Scholarship Scholarship Scholarship 
offered used offered used 


*Difference between the treatment group and the control group was statistically significant at the 0.05 level. 

NOTE: Sample size was 571 treatment group parents and 368 control group parents. Sample size was 368 treatment group 
students and 219 control group students. 

SOURCE: Estimated means and impacts were generated from study's regression models as described in appendix section B-3. 
Parent and student surveys for OSP evaluation, 2015-2017. 


Impacts on Parent and Student Perceptions of School Safety 


The OSP legislation indicated that one purpose of the program was to address shortfalls in DC’s 
public-school safety, and it called for the study to look at parent and student perceptions of school safety. 
The annual parent and student surveys included a question asking parents and students to rate the school 


99 66 


as “very safe,” “somewhat safe,” or “not safe.” In this analysis, parent and student responses rating the 


school as “very safe” were compared with the other two response categories. 


'4 The study’s primary measure of satisfaction assessed parent’s general or overall satisfaction with their child’s school, in line with the focus on 
general satisfaction implied by the SOAR Act’s language. However, parents also responded to a question about satisfaction with 16 aspects of 
their child’s school (for example, academic quality and class sizes). Parents of students in the treatment group were more satisfied than parents of 
students in the control group with 10 of the 16 aspects. Appendix table D-9 presents the full set of these secondary parent satisfaction items. 
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The program had no statistically significant impact on parents’ perceptions of school safety, 
but had a positive impact on students’ perceptions three years after they applied to the program. 
The proportion of parents indicating their child’s school was “very safe” was 65.9 percent for those 
offered scholarships and 62.1 percent for those not offered scholarships (figure 5). Scholarship use also 
had no statistically significant impact on parents’ perceptions of school safety. The proportion of students 
indicating their school was very safe was 60.5 percent for those offered scholarships and 48.7 percent for 
those not offered scholarships, a nearly 12 percentage point difference that was statistically significant.‘ 
The positive impact of scholarship use on students’ perceptions of school safety was 16.8 percentage 
points. 


The impacts on perceptions of school safety for the subgroups were similar to the impacts for 
parents and students overall. There was a statistically significant positive impact on parent perceptions 
of school safety for only one of the eight subgroups: parents of students who were above average in 
mathematics at the time of application (appendix table C-6). For four subgroups of students—those 
attending schools in need of improvement, entering a secondary grade, below average in reading, and 
below average in mathematics at the time of application—the program had statistically significant 
positive impacts on their perceptions of school safety. Findings were not statistically significant for the 
remaining four student subgroups (appendix table C-7). 


Figure 5. Impacts on parent and student perceptions of school safety (percentage rating 
school as very safe) for scholarship offer and use, in third year 


B Treatment 
®@ Control 


Percent 


as Parent Student 


Impact: Impact: Impact: Impact: 
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3.8 4.8 
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40 . 
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Scholarship Scholarship Scholarship Scholarship 
offered used offered used 


*Difference between the treatment group and the control group was statistically significant at the 0.05 level. 

NOTE: Sample size was 504 treatment group parents and 360 control group parents. Sample size was 364 treatment group 
students and 220 control group students. 

SOURCE: Estimated means and impacts were generated from study's regression models, as described in appendix section B-3. 
Parent and student surveys for OSP evaluation, 2015-2017. 


'S Tn addition to overall ratings of school safety, students responded to secondary questions about the frequency of eight specific safety-related 
incidents at school. Students in the control group were more likely to report being threatened with physical harm at school or being bullied at 
school (appendix table D-10 presents the full set of survey items). 
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Impacts on Parent Involvement in Education 


The SOAR Act called for the study to look at the program’s impacts on parent involvement in 
education, which has been linked to better academic achievement and fewer behavioral problems for 
students (Jeynes, 2005; El Nokali, Bachman, & Votruba-Drzal, 2010). Parent survey items measured 
involvement with education at school and in the home. !® Parents reported how often during the school 
year they engaged in activities such as communicating with teachers, or attending school activities or 
meetings. Parents also reported how often during the past month they did education-related activities with 
their child at home such as helping with homework or talking about experiences in school. 


Overall, the program had no impact on parent involvement in education at school or at home 
three years after applying to the program. The number of school involvement events was 22.2 for the 
treatment group and 22.3 for the control group (figure 6). The number of education-related events at home 
was 17.9 for the treatment group and 18.2 for the control group. Differences were not statistically 
significant for parents of students offered the scholarship or students using the scholarship. 


In general, there were few impacts on parent involvement for the study’s subgroups. 
However, for one subgroup—parents of students who were in elementary grades when they applied to the 
program—there were statistically significant negative impacts on their involvement in education at both 
their child’s school and in the home (appendix tables C-8 and C-9). Also, for parents of students who 
were in secondary grades when they applied to the program, there was a statistically significant positive 
impact on their involvement at school (appendix table C-8). 


Figure 6. Impacts on parent involvement in education at school and at home (number of 
events reported) for scholarship offer and use, in third year 


@ Treatment 
Events @ Control 
30 At school At home 
Impact: Impact: 
-0.1 -0.1 


20 22.2 222) 
10 
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NOTE: Sample size for school involvement was 474 treatment group parents and 345 control group parents. Sample size for 
home involvement was 506 treatment group parents and 361 control group parents. 

SOURCE: Estimated means and impacts were generated from study’s regression models, as described in appendix section 
B-3. Parent surveys for OSP evaluation, 2015-2017. 


'6See appendix section D-8 for more information about the parent survey items. 
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Limitations 


The study faced challenges in obtaining high rates of response for the testing requirement and for 
the student survey three years after students applied to the OSP, a common problem for evaluations that 
follow individuals over time. However, analyses included in the appendix, taken together with the study’s 
methods for estimating impacts, provide confidence in the findings reported above. First, sections D-1 
and D-2 of the appendix show that the background characteristics of students who provided data for the 
current report mirrored the full sample of students the study attempted to survey and test. Response rates 
differed for students offered and not offered scholarships, which would be particularly problematic if 
achievement at the time of application—a strong predictor of later achievement—was related to whether 
or not students were tested. For example, if students with lower achievement at the time of application 
were less likely to respond and fewer control group students responded, this would create a “biased” 
sample of higher performing control group students. However, students’ achievement scores when they 
applied to the program were not related to whether or not they completed testing (appendix section D-1).'’ 


3. Change in Achievement Impacts Over Time 


Combined with findings from the evaluation’s previous two reports (Dynarski et al., 2017; 
Dynarski et al., 2018), the findings here provide information about the OSP’s impacts over the three years 
since students applied to the program. The program had no effect on reading achievement in any of these 
years. However, for mathematics, negative impacts reported in the first two years were not found in the 
third year.'* Understanding what might have contributed to the change in mathematics impacts between 
students’ second and third years, a finding that has been noted elsewhere (e.g., Mills & Wolf 2017b), is 
the focus of the questions and exploratory analyses below. Because the focus is on examining change 
over time, each analysis uses the group of students who completed testing each and every year it was 
carried out. 


Did faster growth for the treatment group or slower growth for the control group result in the 
two groups performing similarly by year three? 


The treatment and control groups were performing similarly by year three, when a year earlier the 
control group had been doing better. Distinguishing if this change was due to the control group losing or 
the treatment group gaining ground is important if the goal is to understand how voucher programs work. 
For example, some research suggests that students who move from public to private schools can initially 
struggle with the new environment, but may experience achievement gains in later years that allow them 
to catch up with their peers (Mills & Wolf, 2017a). This theory would be supported by finding that the 
treatment group’s achievement improved substantially—and more than the control group’s—between the 


"7 As discussed in the appendix (section D-1), the study did find that students in the treatment group were less likely than students in the control 
group to be located in a DC school in the third year. However, followup analyses did not find a relationship between students potentially moving 
out of DC or dropping out of school, and achievement at the time of application. This, along with additional analyses reported in the appendix, 
suggest that the difference between the treatment and control group in terms of moving, should not have shaped the mathematics and reading 
achievement findings reported in chapter 2. 

'8 The variation in impacts across years, as well as the change in impacts on mathematics achievement between the second and third years, was 
statistically significant (see appendix section D-3). 
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second and third years. Alternatively, the similar academic performance of treatment and control group 
students in the third year may have had nothing to do with students offered vouchers and be more about 
declines in mathematics test scores among students in the control group. 


Students in the treatment group caught up to their peers in the control group through a 
combination of faster growth by the treatment group and slower growth by the control group 
(figure 7). For both groups, average mathematics scores increased between the second and third years, 
indicating that mathematics achievement benefited from an additional year of schooling. But the growth 
in mathematics achievement accelerated for the treatment group (i.e., achievement grew by more than it 
had between the first and second years after OSP application) and decelerated for the control group. 
Larger increases in average scores for the treatment group than for the control group closed the gap in 
achievement between the second and third years. This provides some evidence for the hypothesis that 
students moving to private schools might struggle, experiencing temporary declines in academic 
performance that dissipate over time. But it also raises questions about what happened to students in the 
control group, the majority of whom were attending public schools. 


Figure 7. Average test scores in mathematics for students in the treatment and control 
groups, by year 
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NOTE: Average scale scores were calculated based on the longitudinal sample of 672 students who were tested in 
mathematics at baseline and all three followup years. 

SOURCE: Estimated means were generated from the study’s regression models for the impact of the scholarship offer, as 
described in appendix section B-3. The treatment and control means for each year were regression-adjusted to account for 
baseline differences, and evaluated at the sample mean across both groups. The study administered the TerraNova Third 
Edition, reading and mathematics tests. 


Could the treatment group’s increased achievement growth and the resulting lack of 
achievement impacts in the third year be related to declining use of the scholarship over time? 


The negative impacts one and two years after application suggest that using a scholarship led to 
lower mathematics achievement, at least initially. If participating in the OSP decreases mathematics 
achievement, then impacts could improve if scholarship use dropped significantly in the third year and 
more students returned to DC public schools. Alternatively, if students who were offered scholarships but 
did not use them performed worse than those using a scholarship, including these “non-users” when 
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calculating treatment group achievement scores may bring down the treatment group average and may 


have unfairly masked a positive program impact after three years. 


It seems unlikely that improved mathematics achievement for the treatment group between 
the second and third years and the lack of mathematics impacts in the third year were simply due 
to fewer students using an OSP scholarship (figure 8). Scholarship use for those in the study sample 
declined slowly and steadily over the three years, though a substantial share of treatment group students 
were still using their scholarships in the third year (68 percent). However, mathematics achievement 
impacts first became more negative and then approached zero by the third year. The difference in patterns 
suggests that a change in scholarship use probably does not explain the change in program impacts. 


Figure 8. Percentage of treatment group students using the scholarship, and 
mathematics achievement impacts in scale scores, by year 
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NOTE: Sample size was 423 treatment group students who were tested in mathematics at baseline and all three followup years. 
SOURCE: Scholarship use rate was calculated from payment files from Serving Our Children. Estimated impacts were 
generated from the study’s regression models for the impact of the scholarship offer, as described in appendix section B-3. 


At the same time, there is little evidence that the treatment group would have performed even 
better if scholarship use rates had been higher. When comparing mathematics achievement for those in 
the treatment group who used scholarships and those who did not, the difference in scores in each of the 
three years after applying to the program was not statistically significant (figure 9). This suggests that 
including non-users in the treatment group average did little to change the study’s findings, and non-users 
were not masking a true positive impact of the program. 
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Figure 9. Average mathematics test scores for treatment group students, by scholarship 
use and year 
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NOTE: Average scale scores were calculated based on the longitudinal sample of 423 treatment group students who were 
tested in mathematics at baseline and all three followup years (347 who used the scholarship in Year 3, and 76 who did not 
use the scholarship in Year 3). Baseline scores for the two groups were significantly different at the 0.05 level. Differences for 
Year 1, Year 2, and Year 3 were not statistically significant. 

SOURCE: Estimated means were generated from the study’s regression models for the impact of the scholarship offer, as 
described in appendix section B-3. The treatment and control means for each year were regression-adjusted to account for 
baseline differences, and evaluated at the sample mean across both groups. The study administered the TerraNova Third 
Edition mathematics test to students participating in the OSP evaluation. 


Could the control group’s slowing achievement growth and the resulting lack of impacts in the 
third year be due to the frequency of their changing schools in the study’s later years? 


The hypothesis that public schools produce smaller learning gains than private schools does not 
explain why the control group’s mathematics achievement grew faster in the first two years than between 
the second and third years. One question is whether or not changing schools, which some hypothesize 
disrupts achievement (Schwartz et al., 2017), might contribute to the control group’s slowing growth. 
Specifically, it seems possible that control group students would be changing schools more than treatment 
group students two years after applying to the OSP, and still searching for suitable schools since they had 
not been offered scholarships or the opportunity to attend a private school. If so, additional mobility after 


two years might slow mathematics achievement gains. 


There is little evidence that school changes are a factor in the slowing mathematics 
achievement gains of the control group. Among the control group, mobility rates were comparable two 
and three years after students applied for scholarships. These rates were also quite similar to those among 
the treatment group (figure 10).'!° This suggests that school mobility probably did not contribute to the 


trend in mathematics achievement impacts. 


'° The study also examined the number of times students changed schools over the three years to see whether there were differences between the 
two groups. Control group students were less likely to change schools more than once (29 percent) compared with students in the treatment group 
(36 percent) (see appendix section D-5). 
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Figure 10. Percentage of students in the treatment and control groups who changed 
schools, by year 
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*The difference between the percentage of treatment and control groups changing schools between baseline to Year 1 was 
significant (p<.0001). 

NOTE: Sample size was 423 treatment group students and 247 control group students in the longitudinal sample who were 
tested at baseline and all three followup years. 


Could the change in mathematics impacts be due to the differences in how many students and 
which students participated in testing each year? 


Finally, it is important to make sure that the trends in mathematics impacts are valid and do not 
stem from some aspect of how the study was carried out. As discussed earlier, not all students participated 
in the study’s testing each year. It is possible that differences in which students participated each year 
might contribute to the change in mathematics impacts. 


It is unlikely that the shift in mathematics impacts between the second and third years was a 
consequence of differences in who was tested over time. Analyzing mathematics impacts one, two, and 
three years after OSP application for the set of students who participated in testing in all three years 
shows the same pattern of results as for all students in the study sample: negative mathematics impacts 
after one and two years and no impacts after three years (appendix D-3). 


4. Implications of This Final Report’s Results 


Three years after students applied to the OSP, those offered scholarships were doing no better 
academically but also no worse than otherwise similar peers not given the same opportunity. While it was 
important to explore potential reasons that impacts found in the study’s earlier years had changed over 
time, it was also important to put the final report’s lack of achievement impacts into context and consider 
future directions for the program and continuing assessments of it. 


The current OSP’s impacts on achievement are consistent with those of other recently studied 
voucher programs, and suggest that following students beyond three and four years may be 
important. Recent studies of state voucher programs in Louisiana, Indiana, and Ohio (Abdulkadiroglu, 
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Pathak, & Walters, 2016; Mills & Wolf, 2017b; Waddington & Berends, 2017; Figlio & Karbownik, 
2016) all indicate that students offered or using vouchers tend to score worse on mathematics tests after 
one and two years, when compared to similar peers not using vouchers. The effects on reading 
achievement have been mixed with some, but not all, studies finding negative impacts. Importantly, a few 
of these studies have found that the negative achievement impacts lessen over time, so that students 
offered or still using vouchers after three and four years no longer perform worse than similar peers (Mills 
& Wolf, 2017b; Waddington & Berends, 2017). 


The current evaluation of the OSP and these other voucher studies all raise questions about what 
happens five, six, seven, and more years after students are offered vouchers. For programs with negative 
impacts even after three years, might that change in later years? For programs like the OSP, might 
positive impacts eventually emerge? The recent 2017 reauthorization of the SOAR Act mandated that the 
current evaluation discontinue its work and that the U.S. Department of Education begin a new study, 
with a different methodology.”° Therefore questions about the longer-term impacts of participating in the 
OSP on academic achievement will remain unanswered. Fortunately, the law includes a new requirement 
to use the current evaluation sample to evaluate the impacts of the OSP on college enrollment and 
attainment. This analysis, expected in 2026, will be an important indicator of the program’s long-term 
effectiveness. 


It seems possible that the OSP’s lack of effectiveness might be related to DC families’ 
already-widespread access to school choice. This study examined the impact of being offered a private 
school voucher in a context where families already have the option to apply to a large number of both 
charter and traditional public schools other than their assigned neighborhood schools. An earlier report 
from this evaluation showed that parents of most students in both the treatment and the control groups had 
chosen a school other than their child’s assigned neighborhood school. That report demonstrated that 
choosing schools was associated with parent satisfaction with schools and hypothesized that choice might 
be a reason that most parents in the study—treatment and control group alike—were satisfied (Dynarski 
et al., 2018). Likewise, it is possible that there were no achievement impacts because students in the 
control group were able to select schools meeting their academic needs, even if they were not 
participating private schools. 


Public school choice is expanding nationwide, beyond DC. For example, between the 2009-10 and 
2015-16 school years, the number of magnet schools increased by almost 50 percent and the number of 
charter schools grew by more than a third (U.S. Department of Education, 2017). Future studies of 
voucher programs may find that effectiveness varies depending on the other options available to students 
eligible for the programs. 


Finally, there are signals that the program might be improved. Based on data collected and 
validated by the evaluation, from 2012 to 2014 the percentage of students offered scholarships who chose 


20 See https://www.congress.gov/115/plaws/publ3 1/PLAW-115pub131.pdf for the SOAR reauthorization Act (P.L. 115-31, sec. 908). The law 
specifies that a new evaluation be conducted using an acceptable quasi-experimental research design for determining the program’s effectiveness 
that does not use a control group of students who applied for but did not receive scholarships. 
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to use them to attend a private school the following year steadily decreased.”! According to records from 
the program operator, this trend has continued (figure 11). 


Figure 11. Percentage of students eligible for and offered DC OSP scholarships using a 
voucher to attend private school one year later 
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SOURCE: For 2012-13, 2013-14, and 2014-15 scholarship use rate was calculated from payment files obtained from Serving 
Our Children. For later years, Serving Our Children provided use rates directly. 


Additionally, after using a scholarship for one year, many students discontinue use and enroll in a 
public school.” Exercising choice includes being able to decide whether and when to enter and leave a 
selected school. However, the declining rates of initial and continuing scholarship use could be signaling 
a problem with the set of private schools available to families or the supports the families receive as they 
apply and adjust to schools. Alternatively, the simplified process for applying to DC charter and non- 
neighborhood public schools might contribute to declining use.”? These issues will be explored in a study 
of the OSP’s implementation that will begin in 2019. 


?! These were the years when the program operator used lotteries to offer scholarships to newly eligible applicants who became part of the 
evaluation sample. 

22 Twenty percent of students who used their scholarship in the first year discontinued scholarship use in the second year. 

3 The first year of the DC common lottery—which allows families to apply to any participating public school using one online application—took 
place in 2014, with students matched for the 2014-15 school year. Each year since, the number of participating schools has grown. 
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